Method of determining a similarity transformation between first and second coordinates of 3D features

ABSTRACT

The invention is related to a method of determining a similarity transformation between first coordinates and second coordinates of 3D features, comprising providing a first plurality of 3D features having first coordinates in a first coordinate system which is associated with a first geometrical model of a first real object, wherein the first plurality of 3D features describes physical 3D features of the first real object, providing a second coordinate system, providing image information associated with a plurality of images captured by at least one camera, for each respective 3D feature of at least part of the first plurality of 3D features, wherein the respective 3D feature is captured by at least two of the plurality of images, determining camera poses of the at least one camera in the second coordinate system while the at least two of the plurality of images are captured, determining for the respective 3D feature a second coordinate in the second coordinate system according to the at least two of the plurality of images and the camera poses, and the method further comprising determining a similarity transformation between the first coordinates and the second coordinates of the at least part of the first plurality of 3D features, wherein the similarity transformation includes at least one translation, at least one rotation, at least one scale and/or their combinations in 3D space.

BACKGROUND

The present invention is related to a method of determining a similaritytransformation between first and second coordinates of 3D features,which comprises providing a first plurality of 3D features having firstcoordinates in a first coordinate system which is associated with afirst geometrical model of a first real object, with the first pluralityof 3D features describing physical 3D features of the first real object,providing a second coordinate system, and providing image informationassociated with a plurality of images captured by at least one camera.

Such method may be used, for example, in relation with Vision basedSimultaneous Localization and Mapping (SLAM), such as disclosed inDavison, Andrew J., et al., “MonoSLAM: Real-time single camera SLAM,”Pattern Analysis and Machine Intelligence, IEEE Transactions on 29.6(2007): 1052-1067, which is a well-known technology for creating ageometrical model of a real environment using one or more cameraswithout requiring any pre-knowledge of the environment. The geometricalmodel that has at least depth information is also referred to as a 3Dmap of the real environment. The creation of the geometrical model ofthe environment is also called the reconstruction of the realenvironment. The created (or typically called reconstructed) geometricalmodel could be represented by a plurality of 3D features, such as pointfeatures and edge features. The 3D features describe physical 3Dfeatures of the real environment. A real environment may also be calleda real object, or may be understood to comprise one or more realobjects.

The geometrical model (or 3D map) of the real environment can be createdusing triangulation of 2D observations shared in a plurality of imagescaptured by one or more cameras. The triangulation is a common methodused in 3D reconstruction based on camera images, see Hartley, Richard,and Andrew Zisserman, “Multiple view geometry in computer vision,” Vol.2, Cambridge, 2000.

A pose of a camera describes a spatial relationship or a rigidtransformation between the camera at a particular position and areference coordinate system. The reference coordinate system may beassociated with a real object or with the camera at another position.The spatial relationship or the rigid transformation describes at leastone translation, or at least one rotation, or their combination in 3Dspace.

The reconstructed geometrical model can be used for determining a pose(i.e. position and/or orientation) of a camera. By matching extracted 2Dfeatures of a current camera image with 3D features existing in thegeometrical model a plurality of 2D-3D correspondences can beestablished. Then, the camera position and orientation in a coordinatesystem of the geometrical model can be computed based on thecorrespondences. Camera pose estimation is also known as tracking acamera.

Vision based SLAM facilitates many applications, such as vision basednavigation of a robot system or a vehicle. Particularly, it is apromising technology that could support Augmented Reality (AR) systemsor applications (see Azuma, Ronald, et al., “Recent advances inaugmented reality,” Computer Graphics and Applications, IEEE 21.6(2001): 34-47) in an unknown real environment.

A common problem of various SLAM systems is that a reconstructedgeometrical model of a real environment is up to a scale as anundetermined factor. In this case, the SLAM systems may assign a randomscale. Therefore, reconstructed 3D features have scaled coordinates in acoordinate system associated with the geometrical model compared to truecoordinates as they are in the real world. Further, camera positionscomputed based on the recovered scaled geometrical models are also up tothe scale, see Strasdat, Hauke, J. M. M. Montiel, and Andrew J.Davison,” Scale drift-aware large scale monocular SLAM,” Proceedings ofRobotics: Science and Systems (RSS), Vol. 2, No. 3, 2010.

The undetermined scale factor introduces challenges to determine truecamera movements in, for example, vision based navigation of a robotsystem or a vehicle, and to correctly overlay virtual visual informationto the real environment in an image of a camera in AR applications.

Particularly, in a situation in which multiple geometrical models ofmultiple real objects are created using the same vision based SLAMsystem for tracking the multiple real objects simultaneously, like inCastle, Robert, Georg Klein, and David W. Murray, “Video-ratelocalization in multiple maps for wearable augmented reality,” WearableComputers, 2008, ISWC 2008, 12 IEEE International Symposium on, IEEE,2008, the problem of undetermined scale factors is quite significant.Typically, random scale values are applied to each of the multiplegeometrical models. If the SLAM system switches between the geometricalmodels, the scale may change and, therefore, the user experience incomputer vision applications like Augmented Reality is seriouslyaffected.

A correct scale factor may be applied in defining true sizes ofgeometrical models of real environments or real objects and true cameraposes as they are in the real world.

Various methods have been proposed for determining correct scale factorsthat could define true sizes of reconstructed geometrical models of realenvironments as they are in the real world.

For example, Davison et al. in Davison, Andrew J., et al., “MonoSLAM:Real-time single camera SLAM,” Pattern Analysis and MachineIntelligence, IEEE Transactions on 29.6 (2007): 1052-1067, propose tointroduce calibration objects with known geometrical dimension fordetermining correct scale factors for SLAM systems.

Lemaire at al. in Lemaire, Thomas, et al., “Vision-based slam: Stereoand monocular approaches,” International Journal of Computer Vision 74.3(2007): 343-364 propose to use a stereo camera system to solve theproblem of determining scale factors in SLAM systems. However, using astereo camera is only a partial remedy, since the displacement betweenthe two cameras has to be significant in relation to the distance to theenvironment in order to reliably compute depth of the environment.

Lieberknecht et al. in Lieberknecht, Sebastian, et al., “RGB-Dcamera-based parallel tracking and meshing,” Mixed and Augmented Reality(ISMAR), 2011 10^(th) IEEE International Symposium on, IEEE, 2011,integrate depth information into monocular vision based SLAM to allowcorrectly scaled geometrical model reconstruction by employing a RGB-Dcamera that provides depth information related to image pixels. It ispossible to determine a scale factor from known depth information.However, a RGB-D camera device is not commonly available in a hand-helddevice, e.g. mobile phone, PDA, compared to a normal RGB camera.

Klein et al. in Klein, Georg, and David Murray, “Parallel tracking andmapping for small AR work-spaces,” Mixed and Augmented Reality, 2007,ISMAR 2007, 6^(th) IEEE and ACM International Symposium on, IEEE, 2007,solve the problem of scale estimation by manually defining a baselinebetween a camera capturing two images needed for 3D triangulation.

Sensor fusion with an Inertial Measurement Unit (IMU) could also be usedto estimate the scale, as disclosed in Mitzi, Gabriel, et al. “Fusion ofIMU and vision for absolute scale estimation in monocular SLAM.” Journalof intelligent & robotic systems 61.1-4 (2011): 287-299. One problemwith this approach is the inaccuracy of the sensor values and thereforeexpensive (i.e. calculation intensive) techniques like “KalmanFiltering” or “Bundle Adjustment” are needed to solve the problem.

SUMMARY

It is an object of the invention to determine a correct scale for areconstructed geometrical model of a real environment such that thegeometrical model can be scaled to a true size as it is in the realworld.

According to an aspect of the invention, there is disclosed a method ofdetermining a similarity transformation between first coordinates andsecond coordinates of 3D features, comprising providing a firstplurality of 3D features having first coordinates in a first coordinatesystem which is associated with a first geometrical model of a firstreal object, wherein the first plurality of 3D features describesphysical 3D features of the first real object, providing a secondcoordinate system, and providing image information associated with aplurality of images captured by at least one camera. For each respective3D feature of at least part of the first plurality of 3D features,wherein the respective 3D feature is captured by at least two of theplurality of images, camera poses of the at least one camera in thesecond coordinate system while the at least two of the plurality ofimages are captured are determined, and for the respective 3D feature asecond coordinate in the second coordinate system is determinedaccording to the at least two of the plurality of images and the cameraposes. The method further comprises determining a similaritytransformation between the first coordinates and the second coordinatesof the at least part of the first plurality of 3D features, wherein thesimilarity transformation includes at least one translation, at leastone rotation, at least one scale and/or their combinations in 3D space.

Thus, at least one scale factor may be determined that could be used toscale a plurality of 3D features defined in a coordinate systemassociated with a geometrical model describing a real object bycomputing coordinates of at least part of the plurality of 3D featuresin another coordinate system using, e.g., vision based triangulation.

According to an embodiment, the method further includes transforming thefirst coordinates of the first plurality of 3D features according to thesimilarity transformation.

According to a further embodiment, the at least part of the firstplurality of 3D features contains at least three 3D features.

According to an embodiment, the second coordinates of the at least partof the first plurality of 3D features are determined based ontriangulation according to image positions of the respective 3D featurein the at least two of the plurality of images.

In an embodiment, the method further includes providing a secondplurality of 3D features having third coordinates in a third coordinatesystem, wherein the third coordinate system is associated with a secondreal object or with a second geometrical model of the second realobject, and the second plurality of 3D features describes physical 3Dfeatures of the second real object, and for the respective 3D feature ofthe at least part of the first plurality of 3D features, at least partof the second plurality of 3D features is captured in each of the atleast two of the plurality of images.

For example, the second coordinate system and the third coordinatesystem are the same, and the camera poses of the at least one camera inthe second coordinate system are determined according to image positionsof the at least part of the second plurality of 3D features in the atleast two of the plurality of images and corresponding thirdcoordinates.

According to another embodiment, the second coordinate system isassociated with the at least one camera at a position in which one ofthe plurality of images is captured, and the camera poses of the atleast one camera in the second coordinate system are determinedaccording to camera poses of the at least one camera determined in thethird coordinate system.

According to an embodiment, the method further comprises providing atracking system, wherein for the respective 3D feature of the at leastpart of the first plurality of 3D features, the camera poses of the atleast one camera in the second coordinate system according to thetracking system are determined, wherein the second coordinate system isassociated with the tracking system or associated with the at least onecamera at a position when one of the plurality of images is captured.

According to an embodiment, the at least one camera is at least onesecond camera and the plurality of images is a second plurality ofimages, wherein the method further comprises providing image informationassociated with a first plurality of images of at least part of thefirst real object captured by at least one first camera, and determiningthe first coordinates of the first plurality of 3D features in the firstcoordinate system according to the first plurality of images. The atleast one first camera and the at least one second camera may be thesame or different cameras.

According to an embodiment, the method may further comprise detecting afirst user input, and if the detected first user input is associatedwith a 3D reconstruction operation, performing the 3D reconstructionoperation associated with the detected first user input, wherein the 3Dreconstruction operation comprises executing at least part of the stepsas described in the paragraph above. Further, the method may comprisedetecting a second user input, and if the detected second user input isassociated with a transform operation, performing the transformoperation associated with the detected second user input, wherein thetransform operation comprises executing at least part of the steps asdescribed herein before.

According to an embodiment, the method further comprises detecting athird user input, and if the detected third user input is associatedwith a tracking configuration operation, performing the trackingconfiguration operation associated with the detected third user input,wherein the tracking configuration operation comprises specifying atracking configuration for determining the camera poses of the at leastone camera in the second coordinate system.

According to an embodiment, the first plurality of features comprisespoint features and the second plurality of features comprises pointfeatures.

For example, the first user input, the second user input and/or thethird user input includes at least one user interaction. The userinteraction may be pressing a button, touching a screen, speechrecognition and/or gesture recognition.

According to another aspect, the invention is related to a computerprogram product comprising software code sections which are adapted toperform a method according to the invention when loaded into internalmemory of one or more processing devices. Particularly, the computerprogram product is contained on a computer readable medium and isnon-transitory. The software code sections may be loaded into a memoryof one or more processing devices, such as contained in a mobile device,like a mobile phone or tablet computer, or in a computer device, like apersonal computer or server computer.

A real object may be any object that has a physical geometric shape inthe real world. The real object could further include one or more realobjects. For example, the real object may be an interior of a room or avehicle. The real object could also be, for example, a chair, abuilding, a mountain, a tree or a table. An interior of a room (as areal object) may further include a chair and/or a table. A real objectis also referred to herein as real environment. The real object can alsobe an arrangement of a plurality of real objects. For example, an indoorobject may be an arrangement of a chair, a table, and a sofa.

The first plurality of 3D features represents or describes physical 3Dfeatures of the first real object or at least part of the first realobject. The first plurality of 3D features are, for example, but notlimited to, points, edges, lines, segments, corners and/or any othergeometrical shapes.

The first geometrical model describes the first real object or a part ofthe first real object. At least part of the first geometrical modelcomprises at least part of the first plurality of 3D features. The firstcoordinate system is associated with the first geometrical model. Thefirst geometrical model may be up to a scale compared to a true size ofthe first real object as it is in the real world.

The first coordinates of the first plurality of 3D features in the firstcoordinate system may be obtained from the first geometrical model. Thefirst coordinates and sizes of the first plurality of 3D features in thefirst coordinate system may also be up to a scale compared to truecoordinates and sizes of the corresponding physical 3D features of thefirst real object in a coordinate system of the first real object asthey are in the real world. The scale may be the same or may bedifferent in one, two, or all three directions in 3D space.

The first plurality of 3D features and the related first coordinates maybe generated by using a vision based SLAM method. For example, —anotherplurality of images of at least part of the first real object may becaptured by one or more cameras. At least part of the one or morecameras may be the same as or different from the at least one camera. 3Dfeatures of the first real object may be detected in the anotherplurality of images. Then, correspondences between image projections of3D features in the another plurality of images may be determined. Thefirst plurality of 3D features and the related first coordinates couldbe determined according to the image projection correspondences. Cameraposes of the one or more cameras relative to the first real object whenthe another plurality of images are captured may also be determinedbased on the correspondences. The determined camera poses may be up to ascale compared to true camera poses as they are in the real world. Thefirst coordinates determined in this way may also be up to a scalecompared to true coordinates as they are in the real world.

The first plurality of 3D features may also be created in a 3D animationsoftware, e.g. a CAD program. A real object might be printed by using a3D printer according to the first plurality of 3D features.

The plurality of images may be captured by one camera or multiplecameras. Each of the plurality of images captures at least part of thefirst plurality of 3D features. For example, the respective image maycapture at least part of the first real object that containscorresponding physical 3D features.

A camera is an image capturing device to capture imagery information ofreal objects. Intrinsic camera parameters of the at least one camera maybe provided or calibrated. The present invention can be applied withreceiving image information from any camera providing images of realobjects. It is not restricted to cameras providing color images in theRGB format. It can also be applied to any other color format and also tomonochrome images, for example to cameras providing images in grayscaleformat. The camera may further provide an image with depth data. Thedepth data does not need to be provided in the same resolution as the(color/grayscale) image. A camera providing an image with depth data isoften called RGB-D camera. A RGB-D camera system could be a time offlight (TOF) camera system. The at least one camera could also capturelight that is invisible to human eyes. For example, the at least onecamera may be an infrared camera capturing infrared lights.

The second coordinate system may be associated with a second real objector with a second geometrical model of the second real object. The secondcoordinate system may be different from or the same as the firstcoordinate system. Further, the first real object and the second realobject may be the same, or different, or have at least one common part.

The second coordinate system may also be associated with the at leastone camera at a particular position. The second coordinate system may bedefined as the same as a coordinate system associated with the at leastone camera at a position when one of the plurality of images iscaptured. For example, the origin of the second coordinate system couldbe at the optical center of the at least one camera capturing the one ofthe plurality of images.

Further, based on a camera pose of the at least one camera relative tothe second real object, coordinates or similarity transformationsdefined in a coordinate system associated with the second real objectcan be transformed to a coordinate system associated with the at leastone camera, and vice versa.

According to an embodiment, the at least part of the first plurality of3D features contains at least three 3D features. Each respective 3Dfeature of the at least part of the first plurality of 3D features iscaptured by at least two of the plurality of images.

In one embodiment for determining the camera poses of the at least onecamera relative to the second coordinate system, a second plurality of3D features is provided. Third coordinates of the second plurality of 3Dfeatures in a third coordinate system associated with the second realobject or a second geometrical model of the second real object may alsobe provided. The second plurality of 3D features represents or describesphysical 3D features of the second real object or at least part of thesecond real object. The second plurality of 3D features are for example,but not limited to, points, edges, lines, segments, corners and/or anyother geometrical shapes.

In one example, the second coordinate system may be the same as thethird coordinate system. Then, the third coordinates are also valid inthe second coordinate system for the second plurality of 3D features.

For each respective 3D feature of the at least part of the firstplurality of 3D features, the camera poses of the at least one camerarelative to the second coordinate system when the at least two of theplurality of images are captured can be determined according to at leastpart of the second plurality of 3D features. For this, each of the atleast two of the plurality of images may capture at least three 3Dfeatures of the second plurality of 3D features. For example, therespective image may capture at least part of the second real objectthat contains corresponding physical 3D features.

The camera poses relative to the third coordinate system may bedetermined according to image positions of the at least three 3Dfeatures of the second plurality of 3D features in the at least two ofthe plurality of images and corresponding third coordinates of the atleast three 3D features. Several 2D-3D correspondences based on poseestimation methods can be used to compute the camera poses, see in thisregard Petersen, Thomas. “A Comparison of 2D-3D Pose EstimationMethods.” Master's thesis, Aalborg University-Institute for MediaTechnology Computer Vision and Graphics, Lautrupvang 15: 2750. If thesecond coordinate system is the same as the third coordinate system, thecamera poses are also defined in the second coordinate system.

In another example, the second coordinate system may be associated withthe at least one camera at any position (called as reference cameraposition). For example, the second coordinate system may be associatedwith the at least one camera at a position where one of the plurality ofimages is captured. In this case, a camera pose of the at least onecamera in the second coordinate system when each of the plurality ofimages is captured describes a rigid transformation (translation and/orrotation) between the reference camera position of the at least onecamera and a position of the at least one camera when capturing the eachof the plurality of images.

A pose of the at least one camera at the reference camera position inthe third coordinate system can be computed using the same methodmentioned above. Then, the camera poses of the at least one camerarelative to the reference camera position (i.e. the second coordinatesystem) while the at least two of the plurality of images are capturedmay be computed according to the corresponding camera poses determinedin the third coordinate system.

A rigid transformation between the at least one camera at two positionscapturing two of the plurality of images may also be determined based ona planar homography between image positions of 3D features of the secondplurality of 3D features captured in the two images. This requires thatthe two images capture the same 3D features.

A planar homography exists between the two images between imageprojections of 3D features on a 3D plane. When the 3D features of thesecond plurality of 3D features are located on the same plane in 3Dspace, a planar homography between the two of the plurality of imagescan be estimated. When the 3D features are point features, the planarhomography may be estimated based on at least four point features usingtechniques proposed in Agarwal, Anubhav, C. V. Jawahar, and P. J.Narayanan. “A survey of planar homography estimation techniques.” Centrefor Visual Information Technology, Tech. Rep. IIIT/TR/2005/12 (2005).Other geometric 3D features, such as lines or conics, projected in thetwo of the plurality of images can also be used to determine the planarhomography according to Agarwal, Anubhav, C. V. Jawahar, and P. J.Narayanan. “A survey of planar homography estimation techniques.” Centrefor Visual Information Technology, Tech. Rep. IIIT/TR/2005/12 (2005).Particularly, the planar homography may also be estimated based on atleast two conics features using techniques proposed in Agarwal, Anubhav,C. V. Jawahar, and P. J. Narayanan. “A survey of planar homographyestimation techniques.” Centre for Visual Information Technology, Tech.Rep. IIIT/TR/2005/12 (2005). Then, a vision method based planarhomography decomposition, such as in Malis, Ezio, and Manuel Vargas.“Deeper understanding of the homography decomposition for vision-basedcontrol.” (2007). APA, can be applied to determine the rigidtransformation between the at least one camera at the two positionsaccording to the one planar homography. This may further require thirdcoordinates of the 3D features of the second plurality of 3D features todefine a position of the plane where the 3D features are located.

In another embodiment for determining the camera poses of the at leastone camera relative to the second coordinate system, a tracking systemis provided and the second coordinate system may be associated with thetracking system. Particularly, the tracking system may be a mechanicalarm, and the at least one camera may be mounted to the mechanical arm,like a system proposed in US Patent Publication No. 2010/0239121 A1. Thesecond coordinate system may also be associated with the at least onecamera at a position when one of the plurality of images is captured.

The tracking system may also be an optical tracking system, and the atleast one camera may be tracked by the optical tracking system accordingto fiducial markers mounted to the at least one camera, like a systemdisclosed in Pustka, Daniel, et al. “Spatial relationship patterns:Elements of reusable tracking and calibration systems.” ISMAR 2006.

With using a tracking system, the camera poses of the at least onecamera relative to the second coordinate system can be determinedaccording to the tracking system.

For the respective 3D feature, at least part of the camera poses of theat least one camera while the at least two of the plurality of imagesare captured may be determined previously. For example, another 3Dfeature of the first plurality of 3D features may also be captured inthe at least two of the plurality of images, the camera poses mightalready be determined for the another 3D feature. When this is the case,it is not necessary to determine corresponding camera poses again.

The second coordinate of the respective 3D feature in the secondcoordinate system can be determined according to the 2D image positionsof the 3D feature in the at least two of the plurality of images and thecamera poses of the at least one camera relative to the secondcoordinate system while the at least two of the plurality of images arecaptured.

Correspondences between 2D image positions of the 3D feature in the atleast two of the plurality of images may be determined. Then, atriangulation method, such as in Hartley, Richard, and Andrew Zisserman.Multiple view geometry in computer vision. Vol. 2. Cambridge, 2000, canbe used to determine the second coordinate of the 3D feature from theimage correspondences.

The method may further comprise the following steps:

The similarity transformation can be determined based on the firstcoordinates and the second coordinates of the at least part of the firstplurality of 3D features according to a method as disclosed in, e.g.,Umeyama, Shinji. “Least-squares estimation of transformation parametersbetween two point patterns.” Pattern Analysis and Machine Intelligence,IEEE Transactions on 13.4 (1991): 376-380. The similarity transformationmay include at least one translation, at least one rotation, at leastone scale or their combinations in 3D space. The at least one scale maybe derived from the similarity transformation.

The first coordinates of the first plurality of 3D features in the firstcoordinate system may be further transformed by the similaritytransformation or scaled by the at least one scale. For example, thefirst coordinates of the first plurality of 3D features defined in thefirst coordinate system may be transformed to the second coordinatesystem by the similarity transformation.

Moreover, the invention is concerned with a computer implemented userinteraction method, as described herein, for a user to interact with thepresent invention. A user interaction could be one of speaking, running,jumping, blinking, and/or moving any parts of the user. A userinteraction may also be pressing a button, touching a screen, speakinginto a microphone, gazing or gesturing. The user interaction may also beplacing a real object into a field of view of a camera such that thereal object is recognized based on an image of at least part of the realobject captured by the camera.

A tracking configuration is related to determining camera poses in thesecond coordinate system. For example, the tracking configuration mayspecify using the tracking system or using the second plurality of 3Dfeatures to determine the camera poses of the at least one camera in thesecond coordinate system. When using the second plurality of 3Dfeatures, the tracking configuration may specify an appearance or aphysical size of the second real object containing the second pluralityof 3D features. For example, the second real object may be a visualsquare marker, and the tracking configuration could specify anappearance of the marker to identify the marker based on camera images.The tracking configuration may also indirectly set the third coordinatesof the second plurality of 3D features by specifying a physical size ofthe marker.

The tracking configuration may be previously specified. The trackingconfiguration may also be specified according to a user input. Thecomputer implemented user interaction method may instruct the user toplace a pre-defined object, such as a visual square marker, into a fieldof view of the at least one camera. Placing the marker into the field ofview of the at least one camera may let the tracking configuration bespecified according to analyzing an image of the marker captured by theat least one camera.

BRIEF DESCRIPTION OF DRAWINGS

Aspects and embodiments of the invention will now be described withrespect to the drawings, in which:

FIG. 1a shows a flowchart of an exemplary embodiment of the presentinvention,

FIG. 1b shows a flowchart of an exemplary embodiment of the presentinvention,

FIG. 2 shows an exemplary scene of the present invention,

FIG. 3 shows a flowchart of an exemplary embodiment of potential userinteractions based on the present invention.

DETAILED DESCRIPTION

In the following, aspects and embodiments of the invention will bedescribed with reference to the exemplary embodiments as shown in FIGS.1 to 3. Some of the following embodiments, such as shown in FIG. 3, aredescribed with reference to using a mobile device, such as a mobilephone, but the invention may be applied in principle with any processingdevice, such as implemented in computer devices comprising one or moremicroprocessors for performing the respective steps. The steps asdescribed herein may also be performed by a network of processingdevices, such as a network of computers or a mobile device communicatedwith a server computer. FIG. 1 may refer to the combination of FIG. 1a-1 b.

Referring now to FIGS. 1a-1b in connection with FIG. 2, step 1001provides a first plurality of 3D features of a first real object andfirst coordinates of the first plurality of 3D features in a firstcoordinate system associated with a first geometrical model of the firstreal object. For example, the first real object is an indoor environment2001 that includes a table 2011, a chair 2013, a bed 2012, and a door2014 (see FIG. 2). The first plurality of 3D features, in this example,represents physical 3D features 2003 existing in the indoor environment2001. A possible geometrical model is the geometrical model 2021 of apart of the indoor environment 2001. The model 2021 is up to a scalecompared to the true size of the indoor environment 2001. A firstcoordinate system IS is associated with the model 2021. The model 2021comprises at least part of the first plurality of 3D features. The firstcoordinates of the first plurality of 3D features defined in the firstcoordinate system of the model 2021 is also up to a scale.

As shown in FIG. 2, a mobile device 2030 contains a camera 2031 used tocapture images. The mobile device further has a processing device 2032that can be used to perform any of the steps according to the inventionas described herein. The mobile device also includes a touchscreen 2033that can display a graphical user interface such that a user can touchor press physical or displayed buttons of the graphical user interface.

Referring again to FIG. 1, step 1002 provides a second plurality of 3Dfeatures of a second real object and third coordinates of the secondplurality of 3D features in a second coordinate system associated withthe second real object. The second real object may be the square marker2002 as shown in FIG. 2. The second plurality of 3D features mayrepresent the four corners 2004 of the square marker 2002.

The camera 2031 of the mobile device 2030 (see FIG. 2) is used tocapture at least part of the first plurality of 3D features and at leastpart of the second plurality of 3D features in a plurality of images instep 1003. The first plurality of images contains at least two images.In step 1004, a 3D feature is selected from the first plurality of 3Dfeatures. In step 1005, it is determined if the selected 3D feature iscaptured by at least two of the plurality of images. If not, then go tostep 1012, otherwise, an image is selected from the at least two of theplurality of images in step 1006.

Step 1007 determines if at least three 3D features of the secondplurality of 3D features are captured by the selected image. If not,then go to step 1009, otherwise go to step 1008 and determine a camerapose of the camera relative to the second coordinate system while theselected images is captured according to image positions of the at leastthree 3D features of the second plurality of 3D features andcorresponding third coordinates of the at least three 3D features. Instep 1008, the camera pose might be determined for one of previousselected 3D features of the first plurality of 3D features. For example,the camera pose when capturing the selected image might be determinedfor a previous selected 3D feature if the selected image captures theprevious selected 3D feature, and in this case the camera pose can bedirectly provided. Step 1008 also sets the selected image as areconstruction image for the selected 3D feature. If all of the at leasttwo of the plurality of images are processed (step 1009), go to step1010, otherwise go to step 1006 and select another image from the atleast two of the plurality of images. Step 1010 determines if there areat least two reconstruction images for the selected 3D feature. If not,go to step 1012, otherwise, go to step 1011.

Step 1011 determines a second coordinate for the selected 3D feature inthe second coordinate system. The second coordinate is computed based ona triangulation method using image positions of the 3D feature in the atleast two reconstruction images and the corresponding camera posesrelative to the second coordinate system. If not every 3D feature of thefirst plurality of 3D features is processed (determined in step 1012),then go to step 1003 and select another 3D feature from the firstplurality of 3D features, otherwise go to step 1013.

Step 1013 determines if at least three 3D features of the firstplurality of 3D features have determined second coordinates in thesecond coordinate system. If not, then stop, otherwise, go to step 1014.

Step 1014 determines a similarity transformation between the firstcoordinates and the second coordinates of the at least three 3D featuresof the first plurality of 3D features based on a method as described inUmeyama, Shinji. “Least-squares estimation of transformation parametersbetween two point patterns.” Pattern Analysis and Machine Intelligence,IEEE Transactions on 13.4 (1991): 376-380. Step 1014 determines the atleast one scale value according to the similarity transformation.

FIG. 3 shows a flowchart of an exemplary embodiment of potential userinteractions based on the present invention, particularly a potentialworkflow for a user who may interact with the mobile device 2030 todetermine a similarity transformation.

The steps 3006 to 3016 on the right side of FIG. 3 refer, at least inpart, to the corresponding steps and routines as shown in FIG. 1 andalready described above. The steps 3001 to 3013 shown on the left sideof FIG. 3 are examples of potential user interactions in connection withthe present invention. For instance, a “3D reconstruction button” may bea respective field displayed on the touchscreen 2033 of the mobiledevice, wherein upon touching or pressing this field (i.e. the 3Dreconstruction_1 button or 3D reconstruction_2 button), the respectiveactions as shown on the right side of FIG. 3 are initiated or performed.In this example, the respective actions are related to a reconstructionof a respective real object. In this example, the first coordinatesprovided according to step 1001 in FIG. 1 are determined according to afirst plurality of images captured in step 3003, which is not necessaryif the first coordinates are already known. The second plurality ofimages according to step 3011, in this case, then corresponds to theplurality of images according to step 1003 in FIG. 1, which is then thebasis for the process according to FIG. 1.

The 3D reconstruction buttons may also be given different names. Thisapplies analogously also for the transformation_1 button andtransformation_2 button related to the transformation process asdescribed above. In general, these buttons are examples of respectiveuser input receiving means for initiating one or more steps of themethod according to the invention by user interaction. The inputinformation according to step 3007 is exemplified for a square marker,but can generally be any input information related to a real objectand/or for defining such real object.

Steps 3001, 3004, 3007, 3009, and 3013 are user interactions and therest of the steps in FIG. 3 are performed on a processing device, herethe mobile device. User interactions 3001 and 3004 may be included in afirst user input. User interactions 3009 and 3013 may be included in asecond user input. User interaction 3007 may be included in a third userinput. A 3D reconstruction operation associated with the first userinput may comprise steps 3002 to 3006. A transformation operationassociated with the second user input may comprise steps 3011, 3012,3015 and 3016. A tracking configuration operation associated with thethird user input may comprise step 3008.

Optional steps 3004 and 3005 may be skipped. The 3D reconstruction_2button may not be necessary to be pressed in order to trigger step 3006.Step 3006 may be directly performed as soon as step 3003 completes.Optional step 3007 may also be skipped. The information of the squaremarker may already be stored on the mobile device 2030 and can beprovided without further user input. The information of the squaremarker could be an appearance of the maker for identifying the markerbased on an image of the marker or a physical size of the marker. Step3012 is performed according to the present invention, particularlyaccording to the exemplary implementation shown in FIG. 1. Optionalsteps 3013 and 3014 may also be skipped. Step 3015 may be performed assoon as step 3012 completes. For example, a computer implemented methodperformed on the mobile device could determine if a pre-requirement(e.g. at least three 3D features of the first plurality of 3D featureshave determined second coordinates) is satisfied for determining asimilarity transformation.

Throughout this document it is described that image informationassociated with an image is provided or received. It is known to theskilled person that this may include providing or receiving anyprocessed or non-processed information (version) of an image, part of animage and/or features of an image which allows for feature recognitionand/or pose estimation. The invention does not require providing orreceiving any raw image data. Processing thereby includes any one ofcompression (e.g. JPEG, PNG, ZIP), encryption (e.g. RSA encryption,Schnorr signature, El-Gamal encryption, PGP), conversion to anothercolor space or grayscale, cropping or scaling the image or conversioninto a sparse representation based on feature descriptors, extraction,and their combinations. All these image processing methods canoptionally be performed and are covered by the terminology of imageinformation of or associated with an image.

The invention claimed is:
 1. A method of determining a similaritytransformation between first coordinates and second coordinates of 3Dfeatures, comprising: obtaining a first plurality of 3D features havingfirst coordinates associated with a first geometrical model of a firstreal object; obtaining image information associated with a plurality ofimages captured by a camera; determining camera poses of the camera fromwhich each of the plurality of images are captured; for each of thefirst plurality of 3D features captured by the plurality of images:determining a second coordinate according to the determined cameraposes; and determining a similarity transformation between the firstcoordinates and the second coordinates of the first plurality of 3Dfeatures, wherein the similarity transformation includes at least one oftranslation, rotation, and scale, in 3D space.
 2. The method accordingto claim 1, further comprising: transforming the first coordinates ofthe first plurality of 3D features according to the similaritytransformation.
 3. The method according to claim 1, wherein the secondcoordinate of at least one 3D feature of the first plurality of 3Dfeatures are determined based on triangulation according to imagepositions of the respective 3D feature in the at least two images of theplurality of images.
 4. The method according to claim 1, furthercomprising: providing a second plurality of 3D features having thirdcoordinates in an additional coordinate system, wherein the additionalcoordinate system is associated with a second real object or with asecond geometrical model of the second real object, and wherein thesecond plurality of 3D features describes physical 3D features of thesecond real object; and for each respective 3D feature of the firstplurality of 3D features, at least one 3D feature of the secondplurality of 3D features is captured in each of the plurality of images.5. The method according to claim 4, wherein the camera poses aredetermined in a second coordinate system according to image positions ofthe at least part of the second plurality of 3D features in theplurality of images and corresponding third coordinates.
 6. The methodaccording to claim 4, wherein the second coordinate system is associatedwith at least one camera at a position in which one image of theplurality of images is captured, and wherein the camera poses of thecamera are determined according to camera poses of the camera determinedin the additional coordinate system.
 7. The method according to claim 1,further comprising: obtaining information from a tracking system; andfor each respective 3D feature of the first plurality of 3D features,determining the camera poses of the camera in a second coordinate systemaccording to the tracking system, wherein the second coordinate systemis associated with the tracking system or is associated with the atleast one camera at a position when one of the plurality of images iscaptured.
 8. The method according to claim 1, wherein the plurality ofimages is a second plurality of images, the method further comprising:providing image information associated with a first plurality of imagesof at least part of the first real object captured by a first camera,and determining the first coordinates of the first plurality of 3Dfeatures in a first coordinate system according to the first pluralityof images.
 9. The method according to claim 1, further comprising:detecting a first user input; and in response to detecting the firstuser input is associated with a 3D reconstruction operation, performingthe 3D reconstruction operation associated with the detected first userinput.
 10. A computer readable medium comprising computer readable codeexecutable by one or more processors to: obtain a first plurality of 3Dfeatures having first coordinates associated with a first geometricalmodel of a first real object; obtain image information associated with aplurality of images captured by a camera; determine camera poses of thecamera from which each of the plurality of images are captured; for eachof the first plurality of 3D features captured by the plurality ofimages, determine a second coordinate according to the determined cameraposes; and determine a similarity transformation between the firstcoordinates and the second coordinates of the first plurality of 3Dfeatures, wherein the similarity transformation includes at least one oftranslation, rotation, and scale, in 3D space.
 11. The computer readablemedium of claim 10, further comprising computer readable code executableby the one or more processors to transform the first coordinates of thefirst plurality of 3D features according to the similaritytransformation.
 12. The computer readable medium of claim 10, whereinthe second coordinate of at least one of the first plurality of 3Dfeatures are determined based on triangulation according to imagepositions of the respective 3D feature in the at least two images of theplurality of images.
 13. The computer readable medium of claim 10,further comprising computer readable code executable by the one or moreprocessors to: provide a second plurality of 3D features having thirdcoordinates in an additional coordinate system, wherein the additionalcoordinate system is associated with a second real object or with asecond geometrical model of the second real object, and wherein thesecond plurality of 3D features describes physical 3D features of thesecond real object; and for each respective 3D feature of the firstplurality of 3D features, at least one 3D feature of the secondplurality of 3D features is captured in each of the plurality of images.14. The computer readable medium of claim 13, wherein the camera posesare determined in a second coordinate system according to imagepositions of the at least part of the second plurality of 3D features inthe plurality of images and corresponding third coordinates.
 15. Thecomputer readable medium of claim 13, wherein the second coordinatesystem is associated with at least one camera at a position in which oneimage of the plurality of images is captured, and wherein the cameraposes of the at least one camera in the second coordinate system aredetermined according to camera poses of the at least one cameradetermined in the third coordinate system.
 16. The computer readablemedium of claim 10, further comprising computer readable code executableby the one or more processors to: obtain information from a trackingsystem; and for each respective 3D feature of the first plurality of 3Dfeatures, determining the camera poses of the camera in a secondcoordinate system according to the tracking system, wherein the secondcoordinate system is associated with the tracking system or associatedwith the camera at a position when one image of the plurality of imagesis captured.
 17. A system comprising: one or more processors; and amemory coupled to the one or more processors and comprising softwarecode executable by the one or more processors to: obtain a firstplurality of 3D features having first coordinates associated with afirst geometrical model of a first real object; obtain image informationassociated with a plurality of images captured by a camera; determinecamera poses of the camera from which each of the plurality of imagesare captured; for each of the first plurality of 3D features captured bythe plurality of images: determine a second coordinate according to thedetermined camera poses; and determine a similarity transformationbetween the first coordinates and the second coordinates of the firstplurality of 3D features, wherein the similarity transformation includesat least one of translation, rotation, and scale, in 3D space.
 18. Thesystem of claim 17, further comprising software code executable by theone or more processors to transform the first coordinates of the firstplurality of 3D features according to the similarity transformation. 19.The system of claim 17, wherein the second coordinate of at least one 3Dfeature of the first plurality of 3D features are determined based ontriangulation according to image positions of the respective 3D featurein the at least two images of the plurality of images.
 20. The system ofclaim 17, further comprising software code executable by the one or moreprocessors to: provide a second plurality of 3D features having thirdcoordinates in an additional coordinate system, wherein the additionalcoordinate system is associated with a second real object or with asecond geometrical model of the second real object, and wherein thesecond plurality of 3D features describes physical 3D features of thesecond real object; and for each respective 3D feature of the firstplurality of 3D features, at least one 3D feature of the secondplurality of 3D features is captured in each of the plurality of images.